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Within the beta-binomial Ernmawork far domain-referenced testing, 
the kappamax reliability index. is proposed for binary Gere saan 
‘gavel on test scores, and studied in* teris of variation, 
approximation and sampling fluctuation. A‘fornula is given, for. the 
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peeecesen of kappamax of tests with doubled. lengths. 
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'/ ‘THE KAPPAMAX RELIABILITY INDEX.FOR DECISIONS -- 
| IN DOMAIN-REFERENCED TESTING,. ay: 


: e < 


foe search for a suitable ‘reliability “index for criterion- 


refér eee tests (Hambleton & BOrEee, 1973) seens to have beéa settled 


. vith the kappa index of sakceniant as proposed by Cohen (19603 algo 


/ abe Suamsnathany Hambleton & Algina, 1974). In a recent work, Auynh a 


1 A 17) argues for the assumption of test eavlucavenbildty (de Finetti, 
/ 1937; Hewitt & Savage, 1955; Zimmerman, 1975), and subsequently _ 
: efines the population meee based on this assumption. In another 
; If. rk, Huynh (1976) details tlie ecapubation of mappe on the basis. of 
le data collected Lice one test administration when equivalent test data 
OZ follow a bivariate beta-binomial nodel. 
AJ . , Kappa has been found to change with the cutoff score(s), 
~ af v hence there is no unique kanpa for a given test’ adninistered, to a 


ee 


group .of examinees. Under ‘test’ exchangeability, kappa varies Between 
0 add inclusive. Hence haceeeniikes at ‘iaaat She set of. cutoff 
score(s) at which kappa is maximized. We propose to call the upper’ 
oui of kappa the kappamax (ras) reliability index for the test (and 


ay - 


for the group of examinees) undéy consideration. | es 


‘ This study will deal mainly with aoe based on binary 
RTRPAREL ESTED ONY The ied bain model will ‘be used extensively 
to dasevihe the test score distribution, Such model would assume that 
the test is composed by randomly diving items from an item universe 
"featuring the instructional objectives: under study. This type of test 


is traditionally described as domain-refererited. - + 
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" (See Table 1 for somé typical numerical illustrations. 


EVALUATION. OF KAPP ig 

BETA-BINOMIAL MODEL - UF 

Ya & , 
Several computational procedures for. the kappa reliability index 


‘at a given cutoff score are described at length in Huynh (1976). To 


; * obtain the kappamax, one may compute kappa for every aoe score and 


then identify the maximum value that kappa can reach. (It is assumed, 


ie 
of course, that only nonrandomized classifications ¢ are to be used.) 


As an illustration, © consider the eteuation where a ere: test 


> is administered to a group of denuivates with true ability defined by : 


the beta density with parameters @ = 11.43 and B = 6.80. ‘The values 


of cupcle at the cutoff scores of 1, 2, 3, 4 and 5 are veececetedis 
.034, 095, ..144, .148 and .091, Hence ‘ie value of kappamax is 
M = .148, 3 
» 
The scheme described above is a gure but lengthy way to get at 
kappamax. A fairly efficient algorithm may be formulated by noting 


that (in all situations considered by the author up to sey ate is 


’ F * : ; ; 
an upturn U-shape function of the cutoff score. Moreover/kappamax  , 


is usually reached at a cutoff score very near the test mean score. | 


/ 


“Table, 1 ae | e. 
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Thus the search for kappanax may be confined to a)? scores in the 


neighborhood of the test mean score. The "stepe to be followed are: 


@ Chéose two consecutive cutoff scores c ‘and c+1 near the test 


uf 


mean score, and compute the corresponding Faye indices K(c) and 
* x 


~ 


K(c + 1). 
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i j 
(b) If K(c) is less than K(ctl),. then compute K(c+2). If K(e+1) is 


larger than x (c+2), 2 i the computation stops, and the kappamax is . 


% 
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‘K(ctl). Otherwise, compue K(ct+3), and so on. - 


a: - may be recalled that 


(3) 


a (c) If K(c) is larger than K(ctl), then compute K(cw1). If K(c=1) 


Pat * 


is less than K(c), then the computation stops and-the kappamax is 
KM = K(c). Otherwise, computek(c-2), and so on. 


AN APPROXIMATION FOR KAPPAMAX . 
FOR LONG TESTS 


» 


The computations described in the last section for kappamax are- 


_ Yather tedious when the number n of test items is fairly large. Unless 


computer facilities are available, it seems desirable to seek an 
approximation for this case. . x 


In a theoretical study regarding kappa based:on normality, 


#uytih (1977) proves that/the value of kappamax’is 


MM = (2/n) sin p, (1) 
where p is the traditional reliability. a follows that tappemas 
_ may, “be approximated by replacing p by a suitably chosen quantity — 
representing the neetelarian between two equivalent forns of. the tests. 


” Several possibilities including the correlation p = Qo} [(n-1)/ (ata, 1) 4 
oe , . 


don 1976, Formula (22)) have been tried, (but it turns out that 
nab 


a réaso le apes for kappamax may be obtained as 


= (2/n) eta” 1454» dp), being the traditional KR21 reliability index. 


‘ 


7 a | - y a = 
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where )- is the test: mean gcore and o” is the test variance. When 
sample data are to be wnady 21 ‘may be estinated by tentang yw and 
o by the corresponding sample mean x and sample variance 3” ° to” 


illustrate the appreximation process, consider for’ example the 


- 


; . a Ss ® : ag (4) 


Duncan. data referred to in Huynh: (1976). Here n = 20, x = 12.54 
21" 52, and 


28 F a 
‘and s = 3.05, An estimate for the KR2] may be taken as a 
hence KM * .35. ‘Exact calculations based on the beta-binomial model 


yield the value of .36 for kappamax. 5" 
Table 2 provides moredata on the approximation of kappamax based 

én normality. Three sets of trde ability distributions are selected 

to represent different degree’ of homogeneity and four devata of 

. test lengths are chosen to be n = 10, 20, 30 and 40. Set 1 yields 

. unimodal and fairly homogeneous test A eae Set 2, on ~ 
the other hand, may be taken aa representative of unimodal distributions 
with moderate degree of homogeneity. Finally, Set 3 is a collection 
of bimodal and relatively heterogeneous test score distributions. For 
all the situations under consideration, the i le based on 
aermeety provides kappamax slightly wasllax than the sorreaponding 
true values. However, the absolute error (difference between a true 
kappamax and its approitimate value) never exceeds six units in the - 


second decimal. The mean absolute error stands at .023 at the relative 


error mrareees out at 3.7 percent. The data of Table 2 also reveals 
that the approximation becomes better with more test items. However, 
a even with only ten items, the errors are etill relatively small and 


partiaps negligible for most sivdbtel purposes. 
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Table 2 
8 


FACTORS AFFECTING KAPPAMAX 


°. m5 
Since kappamax, (based on binary classifications) does not involve 


the cutoff score, only the number n of test items and the true ability 


distribution ( reflected by a and B) will have to be considered; 


» ‘ 
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function of n, hence kappamax defined under normality by 


KM = (2/m) sin p, ig also an increasing function. of n, This stat 
should be expected to hold asymptotically: ‘for the beta-binomial test. 
score model, To shed light on the belarier of kappamax based on tegts 


with short or scdérste lengths, several computations! are made and the 


results are compiled in Table 3. The data vividly document. the 
increasing variation of kappamax’ with respect, to ne Moreover the 


- also indicate that the ite of increase becomes stialler as n take 


x 


larger’ values. The trend tg reiniscent of thee one induced by the 
Spearman-Brown formula, A later section of this paper will re 

ad - a way to project eee for tests. of doubled length. f 

Effect of true (or teat) score variability. The relationship betwees! . 
the true (or test) score variability and kappamex may be inferred from 
the data of Table 3. It may be noted that for the first two dissses 
enn score distributions, kappamax varies in the same direction 
with the standard deviation Ogof. the true score (or aes distri- 
bution. this observation holds for each of the ‘test jake levels. | 
As for the third type of bane score alacetbueions, namely ‘those ith, 9 
* two modes located at both extreme ends’ of the score range, e 

vélabionstits between kappamax aed Ogis not vapanty straighfdeward. It 

may be stated, however, that as a rule, kappamax ‘Reade to increase 

along with the bidvee of heterogeneity in the true (or test) score 


ri a aa A A. - 


Table 3 
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SAMPLING DISTRIBUTION OF KAPPAMAX. 


So ‘far we have considered ‘mostly the case iar both parameters 
a and 6 of the beta distribution are known accurately, In most practical 
situations, however, these quantities are to be estimated from sample 
data. There are many way to accomplish this task. Moment estimates for 
a and § may be obtained by replacing the test. mean yp and the test 
3 * standard deviation of _by their corresponding sample statistics x and 


s in. the formulas defining «a and 8. Thus the moment estimates are 


expressed as 


a 
a 


rd + 1/454) X- ‘ (’ 


~ A Aa : 
-a + n/aos -n, s 
¥ x(n - x) 


where 
a4 — (a - ae 


we? 
a 


These quantities are fairly sais to compute. However, in sampling 
studies, it is usually more desirable to consider the maximum likelihood 
(ML) estimates for their statistical inference properties 4 well, 
known at least asymptotically (e.g. when the sample has a reasonably 
large number of cases.) It may be noted that. for large samples, the 
moment estimates and anes a a aka ML seuntarpirts are very 


near each other in most cases. 


An asymptotic statistical inference theory for kappa and kappamax 
is provided in fluynh en), foe alata test score distributions. 
To be speci tia, let mM be the ML estimate for the population kappamax 
KM. Then 1 may be obtained simply by replacing the two parameters | 
a and in the formula defining kappamax by their ML estimates : 
, 


previously ‘mentioned, If the estimation is based on.N subjects, ,then 


. ‘ ° bs) , ‘ : 
4 8 é 2 N ae 


ws 
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(7) Oe. Se 
. / w, * 
: ait at - 2 has an 1 approximate normal sistrabution with zero mean and =; 
. standard deviation o Standard methods for evaluating this constant ve 
may be Found in such text as Rao L875) for example, Charts developed 
by Huynh (1977b) may also. be used for this purpose. _It. follows from oR i 
this discussion that the ML estimate iM is asymptotically unbiased for | 
the population value KM and has a standard ‘error equal to oN. 

The availability of a standard error for i. ‘would serve at least : | | 
two purposes. First, it may be used to establish confidence intervals . 
for kappamax, For example, | tied du the case where N = 100, a = 3,49 
and. 8 = 1, 08. Recall that the moment estimates and their ML counterparts 1% _ 
are very close to each other for large N's, Numerical computations | 
provide the estimated keppanax of 395: with an estimated standard 
error of .699/ (100) = .070, An approximate 95% contidwace interval ~ 
" for the population kappamax may now be taken as .395 + 1, 645 X .070, 
which is the interval from 28 to .51. ‘Gecond; the value of ony may, 
serve as a starting ‘point for decisions regarding the niaber ¥ shades 
| to be included in any appanas estimation. For i a pilot 
adninistration of a 10-iten test to a group of examinees yields the t 


values a = 5 and 8 = 3, It follows that an estimate for o,, is .539. os j ; 


KM 
X : ; 
If a standard error of .02 is ee then the number N of subjects 
should be at least (.539/. 02)? or 726 approximately. 


Table 4 presents some data regarding the variation trend of om 


with respect to a, B and-n, It is clear that for a given-group of ; 
examinees (e.g. for a given set of a and. g), oxy: is a decreasing 
ction of nh. It follows from previous discussions that Ory is also 


a decreasing Funcet on of KM. Across different groups of examinees, 


gee SP as | . %&: 2 


. however, Css and KM rélate to each other in a rather unpredictable . 


manner, << y 
~ ; ge ge a P : 
Table 4 
fo a a ee 
* ‘ on ai 
PROJECTION FOR _KAPPAMAX OF TESTS WITH DOUBLED 


LENGTHS BASED ON BETA-BINOMIAL DATA 
nt Mee 
| Huynh (1977)*shows that under normality, ‘kappa is asymptotically 1 ca 
-a function of ni, n being the umber of test items. This suggests 
that an adaptation of the Spearman-Brown formula might be stated as 


2 Pe pt im ! 
; i. > aE ee Oe 
ve ad im, a: 


where kn ‘is the kappamax index fora short: test and iM, is the 
a dubs index projected for a test’ whose ‘length ish times longer than 
the short test. If the estyinal Spearman-Brown formula were to be used, 
then the predicted meepenrs would be ee 
\ h'km 3 (3) * 
KM, = ay 
er . 1+ (h -1) ka 


-It turnsout surprisinglythat for the 45 beta-binomial situations 


Lo? 


J 


considered as representatives of a wide range of testing conditions, 
Formula (2} always underprojects kappamax’ whereas Formula (3) always 
overprojects this index for tests with doubled i (h = 2). This. 
suggests that the compromised formula | 
| 18/4 (4) 
KM = ~ 
tt ae w/o) 


_ mean absolute error is .013 and the xéintive error (e.g. the ratio of 


‘‘would doa better projection. It fact it era tn all the 45| cases 


a 


& 
prevacusly mentioned, the difference between the true mer of kappamax 


and its prajected value obtained from Bauselon (4) fi g. the absoluts 


error) nevers, gicnads two units of the second aacinal. Overall he 


the absolute error to the true kappamax) averages out at 1,96 percent. 


As may: be seen from Table'5, the errors do not seem to relate to the 


level of kappamax from Which projection is to be made... 


PROJECTION FOR KAPPAMAX OF TESTS WITH DOUBLED 
LENGTHS BASED ON REAL DATA 
The beta-binomial mnnel has, been used in this study 1 mainly because tae 
of its mathematica? simplicity. Real test data, however, rarely conforms 


exactly to any parametric Form, It seems therefore ‘desirable to 


investigate the accuracy of the projection formula (4) using real test 


data, _— i 

The results subsequently presented inthis section are based on th 
resfonses of 582 examinees oe 138-item test. The test items have . 
difficulty indices ranging from .259 to .861 with a mean of »556 anda | 
standard deviation of .142. Three levels of test length, namely n = 10; | 
20 and 40, are considered and thirty independent projections are made - 
for each n. The following steps are taken in each projection. 
(a) Four subtests, each with n/2 itéms are randoml omly selected from the 
138-item pool, and four subscores’ aaa Xp Xs Y, and Y, are computed : 
for each of the 582 subjects. Po 


ay : ’ | 7 11 


(10) 


1 
scores obtained from two equivalent forms of an n-item test, The true 


-*(b) The sums X =X, + X, and Y -¥, + Y, are computed and treated as 


kappamax is then computed usirg the (X,Y). bivariate distribution and 
the average waeeinal distribution of X and Y, 
(c) Two Nsplit-half” kappamax, one based on (Xj, X,) and the other-on 
(Ys Y,) are computed as in the previous step, and then projected via 
Formula (4) using h = 2. The avacube of the two projected kappamax is 
finally considered as the projected kappamax for the n-item test 
described in the prévious step, , 
- Table 6 reports the summary data compiled from the 90 projections. 

an this table, - the slat ees — (Gulliksen, -1950, page 175) expresses: . 
the extent to which the scores Xy> Xo» Y; and Yo have the same ean: 
.the same vavtance,and the same covariance. When this similarity tac 7 
fulfilled, Le is unity. A small value for. this auantity, on the other 
¥ hand, indicates that the four subtests represented by Xy> Bs Y, and Yy 

depart appregiably from equivalence. Projections based -on the adapted 

Spearman-Brown formula would therefore not be expected to provide good 

results when LuyetS too far from unity. | 
The data of Table 6 clearly show that Formula (4) leads to 
reasonably good projections for kappamax of tests with ‘doubled lengths. 
Over the 90 cases under study, the absolute errors average out at .040 . 
and the relative errors do at 6.47%. As sipented: very good projections 
are ciained (with absolute errors not more than 05) whan the split- 
bade ‘praendinns provides reasonably equivalent subtests. As sisborated in 
Huynh (1976), a high degree of subtest equivalence may be secured by 
pairing the items by difficulty before randomly splitting each pair in 


the subtests. ' Gactocee 
Table 6 
ey EE he | : 


. i: : (11) 


. _™ . 
"5, SOME CONCLUDING REMARKS 


Several aspects regarding the kappamax reliability index for binar} 


Classifications ‘are considered in this study, The advantage of the use. 


of kappamax instead of an ordinary kappa lies in the fact that kappamax | oe 3 


‘not a . function of the. cutoff score. Hence kappamax is unique for a test 
given to a group of examinees. In some sense, -kappamax reflects ithe 


‘of 


situation in which a test functions: Led in terms of consisten¢ 
decisions. It should be mentioned that many aspects of statis wy al . 
. decision theory (such as the minimax principle). are actually based on 
the best performance of a procedure over a wide ald of conditions. The 


asymptotic error prescribed for kappamax — be helpful in dueyging : 


« 
~ 
. 


the number of examinees to be included in a kappamax ‘reliability. 
study. +e adapted Spearman-Brown PEcesore reflected by Formula’ ; 
(4) would be useful in approximating the ‘kappamax directly from the | 
test. data without imposing a paranetric form for the. test score | 
distribution, We would like ‘to caution the resder that, the projection 
5 ‘might not be appropriate if it is based on a ‘small amber of subjects. 
As indicated by Huynh (1976) negative values for sauicae Kappa might be 
a result’ in this case ‘and their interpretation is eather messy. 
Finally there is no ‘reason that kappamax mist be defined as the 
maximum kappa over the whole range of test scores. If ‘there are goad’ 
reasons to restrict the range of ‘possible cutoff scores, tn 
appropriate "kappanax" way then be gpfined. It is expected that most of 
_ the properties mentioned for eappemee: as ns cae in ‘this. study would hold 


° 


"for other i cacel based - a ‘limited. range of cutoff scores, 3 


My 2 
. ay ¢ Pr j ry é Z . 

‘ey 4 ie oe , a F - 

4 alg . 

2 + 


* 
ite 
oy 


ot 


: " SMAMINATHAN, H., HAMBLETON, R K. 4 ALGINA, Je. Reliability of 


— Joarnat of Educational ‘uessupeuaae 1974, u, 262-267. oak, 
wt 
r TERY, D. W. Probability spaces, ane spaces, and the axioms -. i 


(12) . . « ’ et : a 


' REFERENCES : ; 


° ° 
COHEN, a, A ecerficient of agreement for nominal scales. "Educational - ~ BY j 


and. Psychological Measurement, 1960, RR 37-46, 


De RINETEN B. La prevision: ‘ses lois logiques, ses resources subjectives. 


Annales: de 1" Institut Henri Poincare, 1937, ds 1-68. - 


GULLIKSEN, H. Theory of mental test saoeaa: New Yprk: John Wiley §& Son, 
1950. . ee Saar ee ee 


" HAMBLETON, R. K. § NOVICK, M. R. Toward an integration of. theory and 


method for criteri n-referenced tests, Journal of Educational 


Measurement, 1975, 40 159-170, |.” we. 
HEWITT, E & SAVAGE, ls J. “Symmetric measures on Cartesian products. , 
Transactions of the American Mathematical Society, 1950, 80, 470-501. 
HUYNH, H. On the reliability of decisions in domain-referenced testing, 
Journal of anaelbat Neasurenent, 1976, Ag, 253-264. baad 
HUYNH, H. ‘Reliability of multiple classifications, Paper presented : at’ 
: _ the. Spring necting of the Pay chomebric Society. Mudra} Hill, New 
~ Jersey, ape 1976, -Also ‘resubmitted to Psychonetrike at the 
“Suggestion of the managing éditor, March 1977, 
HUYNH, H. Statistical inference for the kappa and kappamax reltabitity | 
indices based on the beta-binomial model, Paper to ve presented 
at the annual meeting of ‘the Psychometric Society. poses of 
North Carolina, June 16-17, 1977, 


RAO, i R.: Linear statistical inference and ‘its siplization New at . “ny, 
Sohn Wiley § Sons, 1973. . »  * 


eriterid-referenced tests: A detision-theoretic formulation. 


a of test. shanty Paychonetrike, 1975, 40, 395-412, eee 


et 40°° 


3 w 
‘ , a 
ik, Be t i A 


th, 


). 


, Ph 2 ee SS 
; ae : ok 


Variation of Kappa a8 Function of Cutoff ° 
ge s Score c for n =,10 


_ — _. 
11.43 6.80 6.27 012 .065 ,095 .153 .204 .236 .242 4215 .154 071 
3.09 .98,°7.59 0151 269.358. .426 .476 .512 .333 538 518 437 


2142298 5.54 423.544 .602 .633” .649 .654 .648 .627. .582 .476 
30.30 5.00 .749 809 .832 .843 .847 .847 .843 .832 .809 .749 Ee i 
’ ‘ " % \ 6 : A 2 : 
¥: 
& 
3 Table 2 
‘A Comparison Between Exact Kappatiax And Its ~ 
- ' : Approximate Value Based on Normality 
a@- > B b 5, Exact | Appr. Error. Percent ie 
_ KM KM _ Error 
; 11.43 6.80 10 4354 242.2304 012 H.96Z : 
20.523. ' 361 «= 6350S 0 305 : 
30. 622.436.4277, 0092.06 
_, 405 .687,. 490. 3482 = 0081.63 
i «3.49 1.08 10 .686 .515 * 2481 .034 §=©6.80 
pn s9 20 :814 .630 605 .015 2,38 
a - 30 .868 .689 .669 .020 2,90 
Bi age OP ok 40.897 .727  .709' = .018 2.48 é 


« 80-(.50 10 .885 .744 .692 .052 6.99 
20 939.814. w 776-0038 4,67 
30.959 847° .817 = 030. 3.54 

40.968 «=.866 0S ,.839 027 3,12. 
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o od } eer a 
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ox | Table 3 i we 3 gr 
* Variation of Kappamax as Function of Test Length i. 
= | - * ‘and Zant Score ee a 
Description’ a B- Oo a=5 ©10 - 20: 40 
x: ‘ Unimodal 3.49 1.08 .180 . 2395 .515 4630) 6727. 
- with mode. 99 = 
sy Mot at the 2°00 . 3.00 .161 273.395.523.639 , 
\ extreme = «11.43. 6.80 110-3148) .242 4361 > .490 “4 
“Scores 10.00 10.00° 109. 6142-224. 4341 471 i: 
dn Unimodal 1.00 1,00 .289 -.567 .670 758 825 . 4 
oh NE OMS 1000 98 TS, BAG SESE TAS a 
(extreme = 3.09 984190 420.538 4650 * 742: 
ian _ a 5.00 1.00 .141 .336 460, .582 .688 
0 — Bimodal «= 3030995791 847 | 891. 922 
with modes 60° 450 346° 688 «5769 038 ead 
~ ™ /. extreme -80 .50 +287 .654 744, “4814-86650 
en +80 20 .200 .710 .785 .844 .889 
t 
. t 
7 A T 
. BA - . , , . 
. Ie. o re i , . 
= 7 ? = . . a ‘ 
Rs w : gost # : ‘ 


Se. Ody »* Cb * 
# t : 
: ‘Table4 =. 
Variation trend of o,,, with Respect to n 
» mown . > 
a 8 ,o, nai. "20 30 | 40. | 
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Table 5 
Projection Errors for Kappamax of Tests With =| “ae 23 
Doubled Lengths Based on Beta-Binomia] Data 
Range of km -No of © Mean of Mean of * 
Proj. . Rel. Error Abs. Error * 
a Sd . 
_ .000- .200, 3 | 6.53% .006 < 
201-400 7 © 1.69 = ——«, 007 ; 
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:801- .999 9 1.34 012 
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Table 6 ; 
" Projection Errors for Kappamax of Tests With © 
Doubled Benghts Based On fen Data 
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